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ABSTRACT 



This paper discusses how educational program developers can 
design useful systems for gathering ongoing information about program 
outcomes to aid in the decision-making process . An introductory section 
offers basic definitions and a rationale for collecting outcomes information. 
Next, roadblocks to using information for decision making are identified. 
These include too much information, a perception of information as 
threatening, and the psychology of using information for making decisions. 

The remainder of the paper elaborates on requirements of an effective system 
for monitoring programmatic outcomes. These include: (1) identification and 

involvement of key stakeholders throughout the process; (2) establishment of 
a program vision; (3) development of measurable outcome goals; (4) 
development of indicators for these goals; (5) identification and 
prioritization of needs; and (6) development of action plans to reduce needs 
and achieve goals. The paper concludes that to be effective, a measurement 
system must produce information useful to teachers and other individuals who 
have the closest contact with students. (Contains 32 references.) (DB) 
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Introduction 



As every educator knows, measuring the results or outcomes of education is being 
emphasized throughout the country. A great deal of time and resources are being put into 
looking at how well schools are meeting goals for student achievement. Most of the efforts 
of Goals 2000 and numerous state educational reform efforts are designed to establish 
educational standards and methods for measuring how well those standards have been met. 



Even though measuring outcomes of the educational process is not a new idea (after all, 
teachers have been measuring student achievement in their classrooms since the dawn of 
formal education), it has been curiously lacking when educators have looked at the overall 
quality of educational programs. Evaluation of educational programs is typically seen as an 
activity to be conducted at the end of some phase of the program, often using outside 
experts. The reasons for evaluation generally relate to compliance with policies, the 
conditions of a funding agency, etc. Further, when outcomes or results are evaluated, the 
efforts tend to focus on specific events, like evaluating conferences with questions such as: 
"Please rate the quality of the keynote speaker," rather than looking at the impact of a 
program - such as whether staff behavior has changed as a result of training, or whether 
student learning is changing. 
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The purpose of this paper is to present a different perspective on the measurement of 
educational outcomes and the impact of the implementation of programs and reform efforts. 
The primary purpose for looking at outcomes should be to provide information that will 
improve the quality of these programs. Issues of compliance and accountability, while 
important, should be secondary in the design of a system to measure educational results. If 
results are not going to be used as a part of the programmatic improvement process, telling 
us where improvements are needed, educators will see little point in gathering the 
information, and a lot of effort will be wasted. This paper is written for educators involved 
in the design, development, and implementation of educational programs. The issues raised, 
however, are relevant for anyone interested in looking at results of the educational process. 
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This paper will present a broad overview of the use of information in decision-making, 
presenting both theoretical issues and some basic practical points. It is not intended to be a 
step-by-step guide to developing an information system for monitoring educational 
outcomes. Indeed, the complex nature of educational systems makes such a step-by-step 
guide difficult, if not impossible, to describe. As will be elaborated throughout this paper, 
effective systems require stakeholder involvement throughout the design, development, and 
implementation of the system. Thus, each system will be different, although the basic 
concepts described in this paper should apply fairly generally to all systems. 



Program evaluation 



In a broad sense, program evaluation is an activity which includes a variety of components: 
assessment of needs, process evaluation (how the program was delivered, whether the needs 
of the audience were met, whether original specifications for the program were followed, 
etc.), and the evaluation of results (outcomes and impact). Of all of these, the evaluation of 
outcomes and impact probably occurs the least, despite the fact that the only purpose in 
developing and implementing educational programs is for the results and impact they will 
have. Affholter (1994) points out that evaluating outcomes is more difficult than evaluating 
the inputs and processes of programs, which undoubtedly contributes to the lack of efforts 
in this area. As was stated above, the purpose of the present paper is to look at one 
component of program evaluation: the measurement of program outcomes (see Rossi and 
Freeman (1993) and Affholter (1994) for more detailed information on this issue). 
Specifically, this paper focuses on how educational program developers can design useful 
systems for gathering ongoing information about program outcomes. For this to occur, it is 
necessary to develop a sensible framework which ensures that the information obtained will 
be useful to the people who need to make decisions, and will actually be used for that 
purpose. 



Definitions 



Before looking at this issue, it is important to clarify several terms. Two terms that will be 
used extensively in this paper are "outcomes" and "indicators." The following definitions 
come from Ysseldyke and Thurlow (1993), and describe the essential meaning for this 
paper: 



Outcomes: Outcomes are the result of interactions between individuals and 
schooling experiences. Results are what can/should/does happen when a person 
has educational experiences. They may be direct or indirect, positive or 
negative, and intended or unintended. 



Indicators: Indicators are symbolic representations of one or more outcomes (or 
inputs, contexts, or processes) that can be used in making comparisons. 
Indicators provide ways of knowing what could/should be looked at to find out 
whether desired results are being reached... (p. 11) 
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As will be described later, this paper will view indicators somewhat more broadly than the 
typical or traditional concept. It should be noted that other terms have similar meaning. 
There is a movement toward using the term "results" in education rather than "outcomes," 
apparently due to some unintended connotations that have become attached to the word 
"outcomes." In the present paper, the words "outcomes" and "results" will be used 
interchangeably. In addition, some authors differentiate between "outcomes" and "impact," 
where impact refers to changes that have occurred in the system as a result of activities, 
while outcomes are the things directly completed or accomplished through those activities. 
For example, developing and disseminating an inservice program might be a program 
outcome, while changes in teacher behavior that occur as a result of going through that 
training would be the impact of the training. For the purposes of this paper, such a 
distinction generally will not be made; all of these will be considered to be outcomes (or 
results). 



Why should we collect outcomes information? 



The most fundamental reason for collecting information on educational results is to help 
educators see what is working and what is not working, so that programmatic improvements 
can be made. Despite such an obvious use for outcome information, there is little evidence 
concerning how these data are used (Sechrest and Figueredo, 1993). Program evaluation 
(which would include the measurement of outcomes) is often seen as a necessary evil, done 
largely (or exclusively) to meet the requirements of the agency that funded the program. 
With that frame of mind, it is easy to see that taking the time to design and conduct an 
effective evaluation ranks fairly low in the priority list of most educators. In particular, little 
thought is given to appropriate ways to measure outcomes, with the implicit assumption that 
if the program is implemented correctly, when good training is done, etc., we will get the 
results we want. Clearly, this is not necessarily true. We have to actually look at the results 
to know whether or not a program is achieving its goals. 



Roadblocks to Using Information for Decision-Making 



Given that most people want to make good decisions, it may seem surprising that people 
don't always use available information, or the best information, for that purpose. There is a 
basic assumption in business that good information leads to good decisions, and that 
managers will use the best information available. That assumption has led to the 
development of increasingly sophisticated management information systems or decision 
support systems. This is another assumption that is often not true. Browne (1993), in 
commenting on the use of information in the decision-making process, states the following: 




And yet, although there is agreement on the importance of information, its role 
in the process is poorly understood. In addition there is substantial, but 
puzzling, evidence that despite the perceived importance of information, it is 
not always used. Or to be more accurate, there is evidence that information 
which is provided through information systems in organizations is not always 
used even when available, (p. 15) 
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In elaborating on why information is not used, Browne (1993) offers the following 
generalizations: 



• Formal information systems are too limited in the type of information they provide, in 
light of the preferences of managers for face-to-face contact and the telephone; 



• Information systems tend to summarize information and lose the richness of that 
information to the point where it becomes bland and unconvincing; 



• The time lag in providing information through formal systems is too great given the 
speed of action required in most managerial decision making; 



• Power and politics in an organization may cause a manager to ignore or distort 
information received; 



• The workload of managers encourages them to be action-oriented and to use very 
current, informal, and unreliable, information; 



• There is a limit to the amount of information which a manager can process in the 
brain; and 



• The brain filters the information it receives and tends to screen out information which 
is at odds with past experience or which creates a cognitive dissonance, (pp. 15-16) 



Thus, simply making information available does not guarantee its use. 



Too much information 



A serious roadblock to the effective use of information is the sheer quantity of it (Browne, 
1993). We live in what is often called the "Information Age." Vast amounts of information 
can easily overwhelm individuals trying to use it. Shuman (1992) states that estimates place 
the amount of information that has been produced in the last 30 years to be greater than the 
total amount produced in the previous 5000 years. Further, the amount of information 
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produced has doubled, and is expected to continue to double, every twelve to fourteen years. 
It becomes virtually impossible for any specialist to keep pace with the amount of 
information available in his or her area of specialty, except perhaps in some of the most 
esoteric fields. This is clearly true for any field of education. As it becomes possible for 
almost anybody to make information available on the Internet, not only will the amount of 
information increase, but the quality will undoubtedly decrease. For example, a search for 
the key words "educational systems change" conducted during the development of this 
paper, using a widely available search engine on the Internet, yielded 1115 documents, most 
of which were not actually relevant or of low quality. Current software does not possess the 
"artificial intelligence" necessary to determine which sources actually deal with the subject, 
let alone evaluate the quality of the document. It requires human intelligence to sift through 
the items found and determine which ones are important. Searches for a topic using fairly 
specific databases devoted to education (such as ERIC) can yield a very large number of 
items, many of which also turn out to be irrelevant. This problem will only increase 
exponentially as the amount of information increases exponentially. 



Given the ease with which vast quantities of information can be stored in computers, and the 
amount potentially available, there is often a tendency in organizations to develop 
information systems that maintain and track over time as much information as possible, 
hoping that some of that information will improve decision-making. Unfortunately, this 
often results in people being overwhelmed by information, and thus not making use of it. 



The "danger” of information 



Another roadblock to using information effectively is that information can be threatening. 
Very often, organizations have used information about problems, not to approach solving 
those problems in a positive manner, but to identify people responsible and punish them. If 
an employee fears that information about outcomes or performance may be used against 
them, they are going to be highly motivated to prevent that information from reaching a 
higher level in the organization, or perhaps even to falsify the information. This is exactly 
the sort of thing that could happen as a result of monitoring educational outcomes, and 
undoubtedly contributes to the lack of motivation to do such evaluations. The fear of having 
information used in a negative fashion, or to control performance, is addressed in Total 
Quality Management (Sashkin and Kiser, 1993). One of the basic principles of TQM is that 
this must not occur. Employees need to feel free to point out problems so that solutions can 
be found and performance of the organization can be improved. 



The psychology of using information for making decisions 



Another roadblock to the effective use of information lies in psychological limits to how we 
can process information. There is an extensive literature on the psychology of 
decision-making and how people use information in making decisions. Some of the key 
early work in this area was conducted by Simon (1955, 1957). Simon argued for a model of 
decision-making known as "bounded rationality." Basically, the argument is that limitations 
of human information processing do not allow decision-makers to operate in a truly 
"rational" manner, and that decision-makers therefore adopt strategies that maximize their 
chances of making a good decision, even though that strategy may not be optimal in a 
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totally rational sense. There are a number of recent reviews which look at a variety of 
models and research in this area (Beach and Lipshitz, 1993; Browne, 1993; Payne, Bettman, 
and Johnson, 1992, 1993). 



However, Orasanu and Connolly (1993) point out that real decisions don't often correspond 
well with the research on decision-making. They argue that the problem is that research in 
this area typically focuses on only a part of the decision-making process, where artificial 
situations are designed which present a fixed set of alternatives to the decision-maker, and 
the researcher then looks at how the decision-maker uses available information to make a 
decision. Real-world situations do not often resemble these laboratory constructions. For 
example, a recent review by Payne, et. al. (1992) discusses the research of Tversky and 
Kahneman (see Kahneman and Lovallo, 1992; Kahneman and Miller, 1986; Tversky and 
Kahneman, 1986; Tversky, Sattath, and Slovic, 1988). A lot of this research has presented 
subjects with different choices, each choice having different probabilities of success and 
different results in terms of what they could gain or lose in an artificially constructed task. 
The problem is that these are artificial situations, and the values that people would bring to a 
real-world situation are simply not there. Educators (or any other decision-makers) do not 
just consider a limited, controlled set of information when they make decisions. The 
information available is constantly changing. There are real consequences for those 
decisions, in terms of what happens to students, teachers, etc. Thus, it's not surprising that 
real decisions do not always seem to make the most "logical" use of information. The basic 
point here is that despite a large amount of literature on the psychology of decision-making, 
there is a great deal we really don't understand when it comes to making "real" decisions. 



Cohen (1993b) argues that the use of a naturalistic paradigm is a more appropriate way of 
looking at decision-making in complex, real-world environments. In elaborating on this 
notion, Cohen (1993a) points out that in a dynamic, real-world environment, decision 
strategies that are not "optimal" in approaching the use of information for decision-making 
may nevertheless result in good decisions. This is because of difference between real-world 
problems and those developed for laboratory experimentation. Real-world problems offer 
continuous, changing information to the decision-maker that can allow for correction and 
changes in the decision. Further, the problem itself may constantly change over time. The 
critical point is that the decision-maker needs to have the appropriate information available 
and use it as the problem unfolds. As Cohen (1993a) states: 



The failure to fully model a problem beforehand is not per se an error; what is 
an error, though, is failing to incrementally improve one's understanding of 
relevant beliefs, preferences, and options as the problem evolves, (p. 57) 



Being able to do this does require that accurate, useful information relevant to the problem 
be available. This is the role that would be met by an information system specifically 
designed to meet programmatic improvement needs. The ways that organizations make 
decisions have clear implications for the design of information systems to support that 
decision-making process (Browne, 1993). 



Developing an Information System to Monitor Educational Outcomes 
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Despite these various roadblocks, effective decision-making should be based on good 
information. Information systems must be designed to meet the needs of the users if this is 
to occur. Implicit in this is that decisions need to be made concerning what information 
should be gathered, how it should be collected, how and to whom it should be disseminated, 
etc. Several recent publications provide valuable advice for organizations on this issue 
(Browne, 1993; Windham, Levin, and Bathory, 1990). Establishing effective, useful 
outcomes information systems may take more time than most organizations realize. It 
requires the active involvement of stakeholders, including beneficiaries of the program 
(Brandon, Lindberg, and Wang, 1993). It involves trust that information will not be used to 
punish people or groups of people. It involves a commitment to the importance of 
improving quality, and a commitment to the time and resources it takes to do that. 



The rest of this paper will elaborate the things that need to occur in developing an effective 
system for monitoring programmatic outcomes. These are essentially the following: 



• Identify and involve key stakeholders throughout the process. 

• Establish the vision for the program. 

• Develop measurable outcome goals. 

• Develop indicators for these goals. 

• Identify and prioritize needs. 

• Develop action plans to reduce needs and achieve goals. 

Although these things are described sequentially, the process is not really sequential. Things 
that happen at one point in the system will affect processes both "before" and "after" it in 
the sequence. For example, results gathered by the system may lead to changes in plans to 
better achieve desired goals. They may also result in revisiting and changing the goals, or 
even making changes in the vision. These results may suggest that other people need to be 
involved, changing the group of stakeholders, who in turn affect all of the aspects of the 
program. Thus, these are more accurately processes that operate in a dynamic, complex 
system, rather than steps to be followed in a specific order. 



Changing and improving educational programs: Stakeholders and vision 



Since the primary purpose of measuring outcomes should be to provide input to the change 
process, improving the quality and effectiveness of our educational programs, we need to 
look briefly at some of the characteristics of change. It is beyond the scope of the present 
paper to discuss all of the complexities involved in effectively changing systems. This is 
described in several recent publications by Michael Fullan (Fullan, 1993; Fullan with 
Stiegelbauer, 1991). Effective change is a slow process, involving both "bottom-up" and 
"top-down" processes. Thus, it is critical that key stakeholders at all levels be identified and 
participate in this process. All too often, change is seen as a reform effort that comes from 
the top-down, with leaders of the effort (at the state or district level) believing that the key is 
to educate the teachers and others about what is to change, and if necessary, to simply 
mandate change. As Fullan points out, this sort of effort is doomed to failure. Individuals 
need to have a sense of commitment to change, with a personal vision and beliefs about 
where they and their organization are going. Effective educational change requires that these 
individual visions merge into a collective, shared organizational vision. It requires that 
schools become "learning organizations," recognizing that change will always occur, that 
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everyone in the organization has a stake in the changes that are occurring, and that everyone 
is a "change agent." 



Despite this environment of constant change and the operation of complex, often chaotic 
forces, progress can be made. The guiding force through this needs to be the individual and 
shared visions of the future. This provides direction for planning, recognizing that plans 
always need to be dynamic and evolving, to effectively respond to changes. What is being 
advocated in this paper is a process for developing outcomes that can be measured which 
are based on the organization's shared vision, so that the organization can monitor where it 
is in its journey. Since that journey will continually shift, our system for measuring progress 
also needs to remain dynamic and in concert with the vision. All too often, information 
systems are established as static systems, which end up providing useless information about 
organizational goals that have long since disappeared. 



Developing outcome goals 



Before an outcomes information system can be developed, the intended programmatic 
outcome goals need to be articulated. In keeping with the previous discussion of vision, 
stakeholders at the various levels of the organization need to be involved in establishing the 
goals for the outcomes of a program. If the development of goals is simply left to program 
managers or grant writers, it is not likely that these goals will reflect the collective vision of 
the people in the organization. In establishing outcome goals, it is important to keep in mind 
that different levels of a system can be impacted by the program. For example, suppose a 
school district initiates a new program designed to deal more effectively with difficult 
student behavior. Some of the outcome goals of such a program might include: 



• Improve the behavior of students with emotional and behavioral problems. 

• Increase the capacity of regular education to meet the needs of students with 
emotional and behavioral problems in the regular classroom. 

• Decrease the number of students with emotional and behavioral problems identified 
as needing special education services. 

• Improve the academic performance of these students. 

• Enhance the knowledge of teachers and other educators in behavior management 
techniques. 

• Improve the attitudes of educators in working with this population of students. 

• Focus efforts on keeping these students in school. 

• Improve the transition of these students into the adult world. 

It should be apparent that there are different "levels" of the system targeted by these goals. 
Some goals are at the level of student behavior (such as improvements in their behavior and 
academic performance); others are at the level of the educator (changes in their knowledge 
and attitudes); still others depend upon complex interactions at several levels (such as 
students staying in school, which would depend upon changes in student behavior, educator 
behavior, and district/school policies in the areas of suspension and expulsion). 



An important part of establishing goals is to be able to answer the question: "What's the 
ideal, but at the same time, what's good enough?" Because program developers often fail 
either to articulate a state of the system they are willing to accept, or expect only the ideal, 
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they either never know when the goals have been reached, or are always dissatisfied because 
they cannot reach the ideal. Outcome goals must be reasonable. In our example, it would be 
unreasonable to expect that all students with emotional and behavioral disabilities would 
behave in the same manner as non-disabled students as a result of this program. Likewise, it 
is unreasonable to expect that all teachers will be able to successfully educate any student in 
their classroom. To establish and expect goals such as these will only lead to frustration. 



Note that describing goals in measurable terms requires that specific indicators of 
performance be defined with desired target values, and this may be difficult. In our example, 
some goals (such as decreased rates of identifying these students as needing special 
education services) are fairly easy to measure, and the indicator that shows how well the 
goal is being achieved is clear in the goal itself. However, the targeted level for this goal 
may not be obvious, and will depend upon information gathered from research literature, 
from the experiences of other districts, and from the consensus of opinion by stakeholders in 
the district. In other words, how much of a decrease should be expected through the efforts 
of the program? Other goals, such as improving the success of these students in the adult 
world, will not only be more difficult to define in terms of targets, but may also be more 
difficult to measure. What are all of the things that would define success in the adult world? 
Employment? Living arrangements? Having friends? All of the above and others? For each 
of these, what target levels or target changes should be established? 



A great deal of work on educational outcomes goals and indicators has been done by the 
National Center on Educational Outcomes (NCEO) at the University of Minnesota; a list of 
various papers they have developed may be obtained on the Internet from their WWW site: 
http://www.coled.umn.edu/NCEO/Default.html. While this work can provide valuable 
information to assist in the development of outcome goals and indicators, it is important to 
remember that the development of any project outcomes and indicators needs to involve 
appropriate stakeholders throughout the design process (Ysseldyke and Thurlow, 1993). 
These goals must be based on their vision for what the program is to accomplish, be 
measurable, and be reasonable. 



All of this goal-setting and planning activity needs to remain dynamic, changing as the 
internal and external environments change. Further, as goals change to meet the changing 
environment, the indicators that need to be measured will change. Thus, the information 
system that will maintain these indicator data also needs to be dynamic. Program developers 
should expect that these things will change, rather than try to build a system and expect it to 
remain constant over time. 



Developing indicators for each outcome goal 



Indicators are data which provide information about how well the goals are being met. The 
data for these indicators are the specific data that would be collected and maintained by an 
outcomes information system. Traditionally, indicators are quantitative sources of 
information, usually expressed as ratios; a number of examples of these can be found in the 
educational outcomes and indicators developed by NCEO (Ysseldyke, Thurlow, and 
Erickson, 1994a-d). In this paper, however, the concept of an indicator will be treated more 
generically, as anything that provides information on how well the intended outcomes and 
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impact are being achieved. Without adequate indicators, we cannot know whether or not we 
have achieved our goals. 



Determining indicators needs to be done by the stakeholders who will actually use the 
information (Ysseldyke and Thurlow, 1993). As stakeholders look at potential indicators, 
they need to ask themselves basic questions about the information they intend to collect: 



• What kind of information will tell us how well we are meeting our goals? 

• Where can we get that information? 

• What will we do with each piece of information when we get it? What does it tell us, 
and what does it suggest for further action? 

• How can we keep the system as simple as possible? 



As was pointed out earlier, there is often a tendency to gather as much information as 
possible, assuming that one will be able to sift through all of the data to find useful 
information. There are a number of problems with this "more data are better" approach: 



• Data collection, recording, and analysis will be time-consuming and thus expensive. 

• Information may be reported back to stakeholders months to years after the data have 
been collected, making them outdated and easy for stakeholders to dismiss, since 
characteristics of the program may have changed during that time. 

• The data may or may not answer the key questions the stakeholders need to have 
answered, since the questions themselves did not drive decisions about what 
information to collect. 

• Stakeholder groups may not have been involved in the design of the information 
system, and thus will feel little ownership. 

In contrast, an effective information system to monitor educational outcomes should have 
the following characteristics: 

• Key stakeholder groups are involved in the design of the system. 

• Outcome goals are developed first, before consideration of the data to collect. What 
results do stakeholders want to see from the program? 

• Decisions about what specific data to collect are determined directly from these goals. 
What kinds of information will provide answers to the question of how well the goals 
are being achieved? 

• Stakeholders know what they will do with the information they receive, since they 
have been involved in determining which information to collect. 

• The system is kept as simple as possible. If possible, data are collected from existing, 
reliable sources, rather than setting up parallel systems. 

• Reports from data collected are provided to stakeholders in a timely manner; the 
stakeholders determine what is needed and when it is needed. 

Using the program example described above, examples of indicators for several of the goals 
might include: 



Goal: Improve the behavior of students with emotional and behavioral 
problems 
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Sample Indicators: 



• Changes in teacher ratings of student behavior. 

• Changes in the percentage of students with emotional and behavioral 
disorders who are subject to disciplinary procedures. 

• Changes in the percentage of these students who are suspended or 
expelled. 

Goal: Improve the academic performance of these students 



Sample Indicators 



• Changes in performance on standardized tests. 

• Changes in student grade-point levels. 

• Changes in teacher ratings of student academic performance. 

Ideally, target values for these indicators would be established, and actual performance 
would be compared to these targets. 



Although it might be tempting for program developers to develop an indicator data system 
themselves, or perhaps also working with a data management professional, such a system 
should never be developed without the involvement of stakeholder groups. If stakeholders 
have not been involved, they will see little reason to use the system for making decisions. It 
is not their system, but rather the program managers' system. Of course, such data collection 
systems are developed this way all too often, and lead to questions from stakeholders such 
as: "Who is going to pay for this?" or "Who is going to collect the data?" The typical 
expectation is that it is the "front-line" stakeholders (such as teachers or support staff) who 
will be asked to do the work. If they have not been involved in the design of the system, and 
see it as functioning for the benefit of some other people, these are reasonable questions for 
them to ask. If, however, they are involved in the design from the beginning, and help to 
develop a system that will truly provide them with information they want, these questions 
will be much less important. If people want information, and the system is designed to meet 
their needs, they will be motivated to gather and use the information. 



Another point that needs to be made, which somewhat complicates the principle of keeping 
the data collection as limited and simple as possible, concerns the human tendency to focus 
only on those outcomes that are measured. If only some valued outcomes are the focus of 
data collection, and others are ignored, the system will be driven to respond only to the ones 
being measured (Affholter, 1994). Thus, it is important to make sure that indicators are 
developed to appropriately measure all key outcomes. 



Although such a system should provide basic information about how well the outcome goals 
are being achieved, it will not tell us why a program is not achieving its goals. Further, it 
does not provide information to stakeholders concerning what they need to do different to 
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better achieve the outcomes they desire. This requires different sources of information, 
which focus more closely on other aspects of the problem, such as how the program was 
implemented, environmental factors that may be affecting it, etc. Failure to achieve the 
goals of an educational program could be due to various problems: the need for better or 
different inservice programs, the need to make curricular changes for students, the need for 
different school policies or procedures, etc. The problem may be outside the school, and 
require a collaborative approach with other agencies and organizations for its solution. Or, 
the goals themselves may be unreasonable and unachievable. Monitoring outcomes provides 
information about results-, it says little about what has gone wrong when the intended results 
are not achieved. It is beyond the scope of the present paper to describe methods of 
gathering this kind of information. However, at the most basic level, teacher-developed 
"action research" (see, for example, Cochran-Smith and Lytle, 1993; Noffke and Stevenson 
(Eds.), 1995) may provide some important insights concerning the implementation and 
functioning of an educational program, which could be valuable in finding ways to improve 
the program. Further, a strong argument can be made for the position that the discovery of 
new knowledge requires the use of qualitative research methods, while quantitative research 
(which typically dominates educational research) does not result in the discovery of new 
knowledge, but in the verification of existing theories (Maykut and Morehouse, 1994). 

Since what is required here is the discovery of why programs are not working as intended, 
qualitative research methods may be called for. 



Identifying the highest priority need areas 



An effective system for monitoring outcomes will not only tell us how well those outcomes 
are being achieved, but also where the needs are. Very often, however, needs assessment is 
seen as something different from assessing the success of a program. It is often done (if 
done at all) in isolation from other data collection about the implementation or results of a 
program. Unfortunately, there also tend to be many extremes of assessing needs - from not 
doing it at all, to gathering so much information that it becomes impossible to understand. 
Further, because the people who are providing that information are typically not doing it 
within the framework being described, people responding to a needs assessment may be 
using different criteria for what they perceive to be a need; they may be looking at different 
levels (student, local, state) but confusing them; they may be confusing what they would 
like with what is truly needed, and they may confuse the desire for resources as a need. This 
last example is probably quite common. People in a school district may state: "We need 
more teachers," when the actual need, at the level of the student, is that they are not able to 
receive sufficient individual attention from their teachers, or that the curriculum is suffering 
because teachers do not have sufficient time to develop it. The solution to this need may 
involve hiring more teachers, but it may also require other things, such as better 
organization of the school day, hiring teachers for curriculum development during the 
summer, involving people other than educators in innovative ways, etc. Clarifying the true 
needs will help program developers find solutions to those needs. 



If we have clearly defined outcome targets for the system, and we have defined the means 
for measuring performance in those target areas, determining the difference between the 
current state and the target state is fairly straightforward. That difference defines a need. 
Thus, the same information system that tells us how the system is doing (the outcome 
monitoring system) also tells us about needs. Once these differences have been identified, 
decisions can be made concerning the most important needs, and what should be done first. 
We need to recognize that we can only effectively work on a few goals at a time. Further, as 
has already been stated, effectively addressing the need will require commitment from 
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people at all levels of the system. Thus, in making plans for action, we cannot tackle 
everything immediately. 



Developing action plans for the identified most important need areas. 



It goes without saying that planning is highly valued in the educational world. Recently, 
many educational organizations have embraced strategic planning as an organizational 
approach to planning, which has resulted in both advantages and disadvantages. The 
positive aspect of strategic planning has been the bringing together of stakeholders within 
the organization to develop their mission and a shared vision, and the recognition that there 
are various factors, both internal and external, which impact upon the organization. There 
have been serious problems, however, with the typical implementation of strategic planning. 
Educational organizations may try to develop plans to cover every aspect of the functioning 
of the system, resulting in an enormous plan that can only serve as a paperweight. People 
often want to "fix" too many problems at one time, resulting in planned activities that take 
more time than anyone can possibly give, and accomplishing very little. Finally, educational 
organizations may lock plans into timelines that don't respond effectively to changes in the 
environment, or they lock the entire planning system into a time frame (5-year plans, annual 
planning retreats, etc.), which forces it to become non-strategic, since the environment is 
not operating under the same rules. Calling a plan strategic, or following a classic strategic 
planning process, does not make the organization strategic in its actual operation. (For a 
good review of the problems with strategic planning, see Mintzberg, 1994). 



What is needed here, in many ways, is much simpler and more straightforward than the 
typical approach to strategic planning. Activities to improve the system need to relate 
directly to making the changes most needed. These would be based at least in part on the 
discrepancies between targeted outcomes and actual indicator data, along with qualitative 
information that suggests the kinds of changes that might result in better performance. Plans 
should be developed to design and implement activities that would most likely cause the 
desired changes. If necessary, changes can and should be made in the information system, to 
make sure the information continues to provide answers to the questions of interest. 



Collecting other relevant information 



It is important to recognize that people can and should use more than outcome information 
in making decisions. Some examples include information about political implications of a 
policy change, availability of resources, new government mandates, etc. All of these will 
help drive decisions. These are all sources of information - they are simply not information 
about the results of the program. Clearly, a comprehensive information system needs to be 
able to provide all sorts of information - about availability of inputs and resources, about 
how the program is being implemented, about the environment, about political implications, 
etc. A comprehensive system also includes the information that comes informally from 
people, both within the organization and outside it. 
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Summary and Conclusions 



Concepts like educational reform and systems change are widely discussed at all levels of 
education, and have resulted in a tremendous amount of activity in recent years. With the 
desire to improve education has come the realization that we have done a mediocre job in 
the past of measuring how successful our educational programs have been in achieving 
desired results. Setting standards for the results we want to see, and investing resources into 
the development of systems to measure success in achieving those standards, has an 
admirable purpose behind it. Unfortunately, as has been described in this paper, this is no 
guarantee that the information will actually be used to improve our educational programs. A 
variety of reasons information is not used well or appropriately have been described. 
Measurement systems must gather information that makes sense to the people who can 
make important changes in the system; this means, in particular, that the information must 
be useful to the teachers and other individuals who have the closest contact with students. 
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